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ABSTRACT 

A two-level model was formulated for the evaluation 
projects engaged in staff development in the Memphis elementary 
schools. The model takes into account that such projects are 
responsible for direct outcome in terms of teacher behavior, and 
indirect outcome in terms of pupil behavior. The two-level model 
provides an opportunity for assessing the validity of the theoretical 
base underlying the project's objectives, and provides for process as 
well as outcome evaluation. (Author) 
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TWO-LEVEL EVALUATION STRATEGY 
FOR PROJECTS ENGAGED 
IN STAFF DEVELOPMENT 
by 

Helen C. Smith 
Memphis City Schools 

This paper presents a model for evaluation of projects engaged in 
staff development which takes into account that such projects are re- 
sponsible for a direct outcome in terms of teacher behavior which is 
hypothesized to bring about a desirable outcome in terms of student 
behavior. 



COMPONENTS OF THE TWO-LEVEL MODEL 

A number of components are considered in the model which are as 
follows: (1) the staff development treatment, (2) administrative support 
to implementation, (3) the teacher outcome in terms of observed implemen- 
tation of the staff development goals, (4) the desired student outcome, 
and (5) background variables expected to influence the teacher implemen- 
tation outcome and the desired student outcome. 
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ANALYSES MADE IN THE TWO- LEVEL MODEL 

As indicated by the above diagram, there are two outcome assessments 
performed. One involves teacher impleinentation of the staff development 
objectives in the classroom (the Level A assessment) and the other in- 
volves the student outcome (LevGl B assessment) • 

Level A Assessment 

Two analyses are performed at Level A, First, scores on instruments 
measuring teachers' classroom implementation are compared between a project 
group of teachers and a control group of teachers for differences of statistical 
significance. This analysis requires that instruments such as classroom ob- 
servation schemes be used so that levels may be objectively quantified. In 
Appendix A of this paper will be found example instruments which yield this 
kind of data on the implementation process. Presented in Appendix B is an 
example of a checklist type of observation form used to evaluate implemen- 
tation which is not suitable for use in this analysis. 

A second analysis is performed at Level A in which step-wise multiple 
regression is used to obtain the rank of the project attendance variable in 
predicting teachers* implementation scores relative to influential variables 
drawn from other components of the model. 

Level B Assessment 

The outcome assessment at Level B involves two analyses also. The first 
compares scores on the student outcome measures from the project group of 
teachers* classes with those of students from the control group of teachers* 
classes for differences of statistical signif icance . 

The second analysis at Level B involves step-wise multiple regression 
equations predicting scores on the student outcome measures. Rank of the 
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measures on teachers' classroom implementation are found relative to 
variables drawn from other components of the model which are expected to 
influence the student outcome, 

SPECIFIC ADVANTAGES PROVIDED BY THE TWO-LEVEL ANALYSES 

1, Prov ide a Safeguard Against Evaluation of a "Non-Event " 
The Level A outcome assessment comparing scores on instruments measuring 
implementation of project objectives in the classrooms of project and control 
groups of teachers enables the evaluator to see whether or not treatment 
levels can be differentiated for a valid assessment at Level B in regard to 
student outcome. In the case of "no difference" results obtained at Level B, 
the evaluator may be saved from making erroneous conclusions about the worth 
of the teacher behavior desired with knowledge of implementation provided by 
the Level A comparison. Charters and Jones in the November, 1973 issue of 
Educational R esearcher point out that differentiation of treatment levels 
is required before valid assessment of effect can be made in terms of student 
outcome, 

2 . Provides Construct Validity Test for Theory Which Underlies Pro.lect 
Objectives in the Face of Poor Im plementation 

The step-vise regression analysis performed at Level B can indicate 
worth of the objectives for teacher behavior in cases where no difference 
appears between project and control groups of teachers in the implementation, 
Level A assessment- The normal variation in scores on instruments measuring 
implementation are assessed for their ability to predict student outcome, 
Provided the instruments appropriately measure the teaching behavior being 
studied, this analysis can provided a validity test for the theory which 
generated the hypothesis that the specified teacher behavior co\ild "bring 
about the student outcome desired. 
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3. Provides an Assessment of the Project Effect in Competition 
with Other Influential Variables 

The step-wise multiple regression analyses performed rank the project 
variables - staff development attendance at Level A and measures of im.- 
plementation at Level B - relative to the component variables selected, 
according to contributions made to equations predicting the two outcomes. 
These analyses provide information regarding effect of the project re- 
lative to say socioeconomic status and can indicate whether or not there 
is a project effect which is over and above that which may be attributed 
to such influences. 

APPLICATION EXAMPLE - MEMPHIS PROJECT READ 

Memphis Project READ is a staff development project funded by the local 
Board of Education. During the 1972-73 school year it provided a week's 
released time for all the system's grade 1 through 3 teachers so that they 
could attend sessions in which lectures and activities v^ere presented re- 
garding diagnostic-prescriptive approaches to teaching reading. After the 
week-long training sessions teachers were accompanied back to the classrooms 
by members of the project staff who assisted in implementation. This per- 
sonnel helped teachers test students to obtain reading skills needs and 
instructional reading levels, demonstrated methods for varying instruction 
and student grouping according to diagnosed needs, and helped set up in- 
terest-centers in the classrooms. 

Level A Instruments 

Two instruments were used to measure outcome in terms of the teachers' 
implementation which were (1) Cohen's Taxonomy Classroom Analysis Sche me 
(TCAS)^ and (2) an individually administered informal reading inventory. 

*Por a detailed description presented by Alan Cohen, see October, 1971 
^ issue of Journal of The Reading Specialist 
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A description of these is presented in Appendix A, The Cohen instrument 
provided a score on extent to which teachers diversified instruction and 
a score on extent to which teachers had students involved in learning 
activities. The informal reading inventories administered to students 
assessed extent to which teachers placed students in reading materials 
at their instructional word recognition and comprehension levels. 
Level A Component Variables 

Variables expected to be influential in regard to teacher outcome 
at Level A which were placed in step-wise regression with the project 
attendance variable were as follows: (1) years experience of teacher, 
(2) number of college methods courses in reading taken by teacher, (3) 
socioeconomic status of the school, (4) pretest level of students, 
(5) ratings of support services and materials by teacher, (6) grade level 
of students, and (7) a written exam over principles presented in the 
project's staff development sessions. 

Level B Instruments 

The instrument measuring student outcome in terms of reading achieve- 
ment were the reading subtests of the California Achievement Tests. 
Level B Component Variables 

Variables expected to be influential in regard to student outcome 
which were placed in step-wise regression with measures on teacher im- 
plementation of project goals included: (1) pretest achievement of 
students, (2) socioeconomic status of school, (3) student attitude toward 
the reading program, (4) teacher ratings of support services and materials, 
(5) years experience of teachers, and (6) number of college methods courses 
taken in reading by teachers . 
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Results 

Comparisons made on outcome measures at both Levels A and B favored 
the project teachers and students over the control groups of teachers and 
students. Heavy loss of student subjects during implementation of the 
City's plan for desegregation placed results of the student comparison 
on shaky ground. It was reassuring to know that the multiple regression 
analyses could let us know whether or not the student outcome results 
should be expected. Results of step-wise multiple regrtission analyses 
performed at the two levels are summarized below. It is seen that the 
project attendance variable made a significant contribution to the equa- 
tions predicting teacher performance at Level A. It is interesting to 
note that the written exam over principles presented in the staff develop- 
ment sessions outranked it. Composite scores on teacher implementation 
of the project objectives at Level A were (:*ound to correlate with student 
achievement at .62 and ranked second best predictor of student achieve^ 
ment at Level B. 
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Another plus associated with the analyses performed is illustrated 
by Project llEAD rc^ultB. Teachers' ratings of support from the school 
Hystein in the form of services and materials were found to correlate with 
stuuent outcome at .52 and were found high ranked in predicting both 
teacher and student performance. Findings like these provide strong 
rationales for requests from project personnel for additional support 
services and materials. 

IMPORTANT CONSIDERATIONS TO BEAR IN MIND BEFORE CHOOSING TO USE THE 
TWO'-LEVEL MODEL 

It is suggested that the following be considered before this model 
is selected for the evaluation of a project: 

(1) There are numerous assumptions involved in the analyses 
performed. Most evaluators will need the assistance of a 
mathematician in performing them and in interpreting results 
obtained . 

(2) The analyses performed provide information regarding a project's 
effect or potential effectiveness. A look at pretest-post test 
gain scores on the student outcome measures could result in a 
significant project effect being judged unacceptable, 

(3) It may be difficult to find the appropriate instruments for 
measuring Level A implementation by teachers. 



APPENDIX A 



INSTRU>CENTS MEASURING LEVEL A TEACHER 
OUTCOME FOR MEMPHIS PROJECT READ 
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Two In.st rumentH were used to obt^nn data on i mp J ementation In 
teachers classroom of Project IlEAD objectives concerning diagnostic- 
prescriptive approaches to teaching reading. The first, Cohen ^s Taxonomy 
Classroom Analysis Scheme (TCAS) , was used to assess extent of individual- 
ization found and extent of pupil involvement in learning activities. In 
using this instrument the observer defines materials and pupils' learning 
activities in terms of an eight-digit Taxonomy coding system. The ob- 
server selects a 42-minute class period and observes a random sample of 9 
students every two minutes as they use materials and engage in learning 
activities. /\ TCAS observation analysis sheet follows. To the left it 
may be seen that A through K treatments were available in the classroom. 
To the right student participation or non participation is recorded. 
(N —non participation, T=transition between tasks) . The score indicating 
student involvement is derived by dividing the number of observations less 
N by the number of observations. 

A second instrument, an informal reading inventory, was used to obtain 
extent to which teachers provided students reading material at appropriate 
word recognition and comprehension levels. Random samples of students 
froa each teacher* s class were asked to read the stories they were to 
have in class the next day. A word-recognition and comprehension score 
was obtained as described on the instrument, a copy of which follows. 
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INDIVIDUAL INVENTORY 
ASSESSING STUDENT ABILITY TO READ AND UNDERSTAND TEXTBOOK 
Name of Student Teacher Grade 
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School 



Name of textbook 



Reading Level 



PART I 



Instructions : Select a passage for reading from the text -- 50 to 100 words in 
length. Ask student to read passage aloud without prereading- As errors occur 
check appropriate classification. 



Word pronounced by examiner 

Word substitution 

Word insertion 

Word omission 



Word order reversal 



_Total_ 
_Total_ 
Total 



Total 
Total 



Total errors 
Total number words in selection 

WORD RECOGNITION SCORE 
(Number of words less number of 
errors divided by number of words) 

TIME 



PART IT Instruction : Read aloud to the student a selection from the text of approximately 
100 words. Upon completion, ask 5 questions concerning the selection. Question 
^1 should call for extraction of the main idea. Questions 2 and 3 should ask for 
recall of factual information specifically stated in the selection- Questions A 
and 5 should involve the making of inference by student- 

Questions Correct 
1. 



2. 



3. 



U . 



3. 



COMPREHENSION SCORE 

(number correct answers divided by number of questions) 



READING HA; ITS afECKLIST 



'□finger i . nc^-if,; G r ..;e repeats words or phrases D vocalizes during silent reading 

U loses plac. d holds book too close to face 



i\PPENDIX B 



E.WIPLE OF INAPPROPRIATE INSTRUMENT 
^^EASURING LEVEL A TEACHER OUTCO>IE 
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>tEMPHIS CITY SCKOOLS 
DIVISION OF RESEARCH 

FLVriNG SCALE FOR EVALUATING CLASSROOM TEACHER 



Directions: Indicate by drawing a line around the appropriate 

number the extent to which a given teacher shows each 

characteristic- Use the following ratings: 

1 - Almost always A - Seldom or never 

2 - Most of the time 5 - Undecided 

3 - Sometime 6 - Not applicable in program 
Instructional Materials and Methods 



1, Knows the skills of reading and how to teach them effec- 
tively, and instruction reflects this knowledge. 12 3 4 5 6 

2, Knows clearly what he is trying to accomplish and 

transmits this knowledge to his pupils. 12 3 4 5 6 

3, Teaches children in materials that will assure their 

success. 12 3 4 5 6 

4, Keeps in the classroom a variety of reading materials 
on levels appropriate for his pupils and encourages 

them to use them. 12 3 4 5 6 

5, Uses reading instructional materials that are interesting 

to children, 12 3 4 5 6 

6, Praises pupils for reading achievement — in terms that 

are acceptable to them. 12 3 4 5 6 

7, Diagnoses constantly and teaches in terms of findings. 12 3 4 5 6 

3. Helps children to recognize their own individual 

strengths and weaknesses in reading. 12 3 4 5 6 

9, Individualizes reading instruction whenever possible. 12 3 4 5 6 

Pupil Attention to the Learning Task 

10, Gets and maintains pupil attention through use of a 

variety of acceptable techniques. 12 3 4 5 6 

11. Involves children immediately in routine activities as 

they come into the classroom each morning. 12 3 4 5 6 

12* Gathers materials for each planned activity before that 
activity is to take place, and has them ready for 

immediate use. 123456 
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